Method and device for coding a mosaic

ABSTRACT

The process for coding a mosaic of images giving a stream of coded data the mosaic being composed of subimages, the subimages being composed of image elements, resynchronization markers in the stream allowing resynchronization of the data at reception level, is characterized in that:  
     the subimages ( 1   1    . . . 1   N ) are placed under one another in the image to be coded so as to give a mosaic,  
     the mosaic is coded element by element according to a TV-type scan so as to provide data of the stream, the first element of each subimage being identified,  
     a resynchronization marker is placed in the stream, before each first element of a subimage, so as to identify the data relating to the subimage.

[0001] The invention relates to a process and a device for coding anddecoding a mosaic of images.

[0002] The field is that of video compression of MPEG type or the like,in the case where this technique makes it possible to implementresynchronization means. It entails for example the H263 or MPEG 4standard or any compression process that offers the possibility ofplacing resynchronization markers at various places in the binary traingenerated during compression.

[0003] In contemporary digital decoders implementing MPEG 2 decoders,the mosaic is generally composed of several pages (screens) each pagecorresponding to a channel (MPEG 2 elementary stream). At any instant, acomplete image is decoded and displayed. Even if only a part of themosaic is useful for display, the mosaic must be completely decoded,thus generating a considerable processing time.

[0004] The aim of the invention is to alleviate the aforesaid drawbacks.

[0005] Its subject is a process for coding a mosaic of images giving astream of coded data the mosaic being composed of subimages, thesubimages being composed of image elements, resynchronization markers inthe stream allowing resynchronization of the data at reception level,characterized in that:

[0006] the subimages are placed under one another in the image to becoded so as to give a mosaic,

[0007] the mosaic is coded element by element according to a TV-typescan so as to provide data of the stream, the first element of eachsubimage being identified,

[0008] a resynchronization marker is placed in the stream, before eachfirst element of a subimage, so as to identify the data relating to thesubimage.

[0009] According to a particular implementation of the process, an imageelement is a macroblock, the coding utilizes the MPEG 2 standard and theresynchronization markers are headers of slices (according to theterminology in the standard).

[0010] According to a particular implementation of the process, an imageelement is a macroblock, the coding utilizes the MPEG 4 standard and theresynchronization markers are headers of groups of macroblocks.

[0011] According to a particular implementation of the process, when thecoding of the subimages is in inter mode with motion estimation, themotion estimation utilizes a window for searching for the motion vectorwhich corresponds to the location of the subimage in the image.

[0012] According to a particular implementation of the process, thecoding of a subimage is forced to intra mode if it has not been coded inthis mode for a specified number of previous mosaic images.

[0013] According to a particular implementation of the process, thesubimages having different sizes, the surface of these imagesconstituting a mosaic support, the mosaic support is also coded and thesupport's coding data are transmitted in the stream at the start of avideo session. If the support varies over time, the support's codingdata are transmitted in the stream at each variation.

[0014] The invention also relates to a process for decoding a mosaic ofimages on the basis of a stream of data comprising resynchronizationmarkers and data for coding mosaics of images consisting of subimages,characterized in that a filtering of these data on the basis of theresynchronization markers and a decoding are performed, to extract fromthe stream and to decode, from among the data relating to the mosaics,only the data relating to selected subimages.

[0015] According to a particular implementation of the process, thedecoded subimages are stored and if a subimage has been coded in intermode, the corresponding stored subimage of the previous mosaic isutilized for the decoding.

[0016] According to a particular implementation of the process, if asubimage may not be decoded, the corresponding subimage of the previousmosaic is taken into account.

[0017] The invention also relates to a decoder for the decoding of amosaic of images on the basis of a stream of data comprisingresynchronization markers and data for coding mosaics of imagesconsisting of subimages, characterized in that it comprises a circuitfor filtering on the basis of the synchronization markers and a decodingcircuit for extracting from the stream and for decoding, from among thedata relating to the mosaics, only the data relating to selectedsubimages.

[0018] The invention also relates to a reader of a video datastreamtransporting data for coding mosaics of images composed of subimages andresynchronization markers, for the displaying of a final image,comprising a decoder for the decoding of the stream and a processingcircuit for the management of the decoder, characterized in that:

[0019] the processing circuit sends the decoder information regardingthe selected subimages useful for the displaying of the final image,

[0020] the decoder comprises a circuit for filtering on the basis of thesynchronization markers and a decoding circuit, for extracting from thestream and for decoding, from among the data relating to the mosaics,only the data relating to the selected subimages.

[0021] The idea of the invention is to utilize resynchronization markersgenerally provided for in the standards to allow resynchronization witha datastream received upon errors or losses of transmission, so as todefine data, in the stream, belonging to various parts of images orsubimages from which a mosaic image is composed. The technique ofresynchronization markers is used to tag the start of each subimage ofthe mosaic and to decode at any instant only the parts of the mosaic tobe displayed.

[0022] The decoding procedure is thus improved in terms of speed ofcalculation, the video decoding of the mosaic of images is accelerated.The aim is to maximally preserve the resources of the central processingunit (CPU) of the reader so as to allow it to perform tasks other thanthose of video decoding of the mosaic. The term “reader” used heredesignates the entire apparatus that comprises the video decoder andalso the whole of the system layer that controls this decoder, as wellas other elements such as audio decoder, graphics processing circuits,display circuits.

[0023] In the case of an MPEG 4 reader the share devoted to themanagement of graphics applications and to the system layer isessential. More and more platforms are equipped with means for decodingMPEG 4 data and it is becoming logical for interactive applications ofthe electronic program guide or EPG type to be implemented in the MPEG 4standard, allowing greater richness at the interactivity level. It istherefore beneficial to render the video decoding as fast as possibleand to simplify the processing of the data in applications of this type.

[0024] This selective decoding is implemented on the basis of aproprietary method at the level of the decoder and on the basis ofinformation originating from the “system” layer that controls the videodecoder. The “standard” decoders not having this improvementnevertheless remain compatible with the technique implemented, althoughwithout benefiting from the advantages of the process proposed.

[0025] Other features and advantages of the invention will becomeclearly apparent in the following description given by way ofnonlimiting example and while referring to the appended figures whichrepresent:

[0026]FIG. 1, a format of mosaic to be coded,

[0027]FIG. 2, a binary train generated by the coder,

[0028]FIG. 3, a processing algorithm of the decoder,

[0029]FIG. 4, a format of mosaic of variable size.

[0030] The proposed process intervenes in three elements of the videocoding/decoding chain: the video coder, the system layer of the readercontaining the video decoder, the video decoder. It utilizes, duringencoding, the technique of video packets, existing for example in thevideo MPEG 4 standard (cf. ISO/IEC JTC 1/SC 29/WG 11 N 2502, annex E,paragraph E.1.1, pages 307 to 308). It consists in placing, in eachimage, resynchronization markers delimiting each video packet, thesepackets being coded independently of one another, as described in thisdocument in paragraph E.1.1 entitled “resynchronization”.

[0031]FIG. 1 represents a succession of three “mosaic” images referenced1, 2, 3 to be coded at the instants t, t+1, t+2. In this example, amosaic image consists of subimages or imagettes of identical size andthe corresponding video therefore consists of the superposition ofsubimages of identical size. Thus, the mosaic image 1 consists of Nsubimages or imagettes referenced 1 ₁ to 1 _(N).

[0032] The binary coding stream for these images is represented in FIG.2. It is a binary train of the elementary stream (to use the terminologyin the MPEG standard), corresponding to the entire mosaic. The videodata relating to the mosaic image are preceded by an image header. Theyconsist of the coded data relating to each subimage. According to theinvention, these data are separated for each subimage by aresynchronization marker. During encoding, the coder places, in thebinary train, resynchronization markers delimiting the video packets,these packets being coded independently of one another, a video packettherefore corresponding to the data between two markers. Each subimageis then processed as a video packet and a resynchronization marker istherefore placed at the start of each subimage, that is to say itprecedes, in the datastream, the video data corresponding to a subimage.In the case of macroblock coding, this marker is placed, in thedatastream, before the first macroblock of each subimage.

[0033] The video coder codes all the video subimages. By placing, forthe coding of the mosaic, the subimages one after another, their coding,by the video coding layer, is effected in a sequential manner, one afteranother when considering the conventional television type mode ofscanning. Thus, by incrementing the synchronization markers from onesubimage to the next, it is possible, when decoding, to tag and toaccess the desired subimage or subimages directly on the basis simply ofthe index number of the associated marker. On the basis of theinformation originating from the system layer indicating the desiredsubimages, that is to say necessary for the production of the finalimage to be displayed, the decoder can therefore initiate the decodingof just these subimages useful for this final image.

[0034] The coding of the subimages can utilize the motion estimation fora coding of inter, predictive or bidirectional type. In order for thesubimages to be decodable independently of one another, it is necessary,when coding a subimage of inter type, for the motion estimation zone tobe limited to a prediction zone corresponding to the location and thesize of the subimage concerned. Thus, a motion vector of the motionestimator is not permitted to point to a subimage other than the onecorresponding to the block with which it is associated. This limitationcan be applied on the vertical axis alone, the standard permitting amotion vector to point outside the image for the horizontal axis.

[0035] To allow the decoder to access the subimages of the mosaic in arandom manner, for example, subsequent to an action by the user, thesubimages must be regularly coded in intra mode.

[0036] Several solutions are possible:

[0037] the coder codes at a relatively high frequency, conventionallyevery 0.5 seconds, the complete mosaic in intra mode.

[0038] if the coding syntax so permits, the coding in intra mode is notperformed on the whole of the image but only on a part of the image.Thus the coder uses this intra mode subimage by subimage in a rotatingmanner. At a given instant, only the first subimages are forced to intramode. The other subimages are coded in predictive mode, hence withtemporal prediction. At the following instant, the subsequent subimagesare the ones that are coded in intra mode, and so on and so forth untilthe last subimages of the mosaic are reached. The procedure thenrecommences for the first subimages. This allows a time dilution of thecost, in terms of bit rate, of the intra-coded subimages and makes itpossible to avoid any bit rate spike that might occur if the coding ofthe complete image were performed in intra mode.

[0039] An implementation much like this second solution consists infurnishing each of the subimages with an intra refresh counter,reinitialized at each intra coding, for example to a value equal to therefresh period Rf, this counter being decremented at each inter codingof the subimage in question. At the start of the coding of the mosaicsequence, once the first mosaic image has been coded in intra mode, thecounters associated with the subimages are initialized randomly between0 and Rf. The intra coding of the subimage is triggered each time thecounter returns to Rf. Moreover, during a change of scene (cut) relatingto a subimage, the refresh counter associated with this subimage may bereinitialized to Rf so as to allow the coding of this subimage in intramode.

[0040] The system layer of the reader implements a proprietary solutionmaking it possible to indicate to the video decoder, from a suitableinterface, which subimages of the mosaic have to be decoded. It sendsfor example the ordered list of index numbers of the subimages to bedecoded. These subimages are exactly those which have to be displayedafter decoding. When a “standard” decoder is used, this information willnot be utilized and the entire mosaic will then be decoded.

[0041] The video decoder receives the binary stream corresponding to thecoded mosaic, as well as the list of index numbers of the subimages tobe decoded. It manages the following information:

[0042] the mode of coding of the current image (intra or predictivemode);

[0043] an indicator assigned to each imagette, signalling whether theimagette has been decoded at the previous instant (subimage_decoded(k)).

[0044] The “subimage_decoded (k)” indicator for k varying from 1 to N,indicator relating to subimage (k), N being the number of subimagesconstituting the mosaic, makes it possible to ascertain whether thissubimage k has been completely decoded at the previous instant and hencereconstructed. Such is the case when this subimage k is coded in intramode or when it is coded in predictive mode (inter mode) and whentherefore the corresponding previous subimage to which the motionvectors refer is decoded, allowing the decoding of this current subimagek.

[0045] The processing algorithm of the reader is represented in FIG. 3.The list, calculated by the system layer, of subimages to be decoded isutilized by the processing algorithm.

[0046] A first step 4 checks whether all the subimages of the list havebeen processed.

[0047] If so, that is to say if we are at the end of the list, the nextstep is step 5. All the subimages necessary for the display of the finalimage having been processed, the decoding procedure is terminated andthe decoded mosaic is returned to the system layer which will process itso as to carry out the display of the final image.

[0048] If not, the next step 6 recovers the index number k of the nextsubimage to be processed, the first subimage if we are at the start ofan iteration. The next step 7 searches for the resynchronization markerk in the datastream. Step 8 checks whether it has been found.

[0049] If it has not, for example if there was an error in thetransmission of the data, we loop back to step 4 to process the nextsubimage, the indicator relating to the current subimage k beingpositioned in such a way as to signal that the subimage could not bedecoded.

[0050] If it has, the next step, which is step 9, checks whether themosaic has been coded completely in intra mode, or else whether subimagek of the previous mosaic image has actually been decoded. This check canbe made on the basis of the positioning of the “subimage_decoded(k)”indicators assigned to the subimages:

[0051] if such is the case, the decoding of the subimage is thenpossible and then, during step 10, the data relating to this subimage kare decoded.

[0052] if such is not the case, the algorithm will check whether thecoding of the macroblocks of subimage k is performed in intra mode, themode of coding of the overall image being the inter mode, however.

[0053] Thus, a step 12 checks whether all the macroblocks of thesubimage have been investigated.

[0054] If they have not, step 14 checks, for the next macroblock,whether it is coded in intra mode:

[0055] if such is not the case, subimage k may not be decoded in intramode, and step 15 positions the indicator in such a way as to signalthat the corresponding imagette has not been decoded. The next step isstep 4 which processes the next subimage.

[0056] if such is the case, the next step 13 performs the decoding ofthe data relating to this macroblock and then step 12 checks whetherthis is the last macroblock to be coded of this subimage k.

[0057] If they have, that is to say, if all the data relating to the,macroblocks of subimage k were able to be decoded, step 11 isimplemented.

[0058] When the data relating to all the macroblocks of the subimagehave been decoded, whether this be on completion of step 13 and 12 orelse on completion of step 10, the subimage is then reconstructed duringthe next step 11 where the “subimage_decoded(k)” indicator is updated.This involves a reconstruction, either on the basis solely of the dataof the stream relating to this subimage at the instant t if the codingof the data was in intra mode, or on the basis of the stored data of thecorresponding decoded subimage of the previous mosaic if the coding ofthe data was in inter mode. The imagette is constructed on the basis ofthe decoded data by performing in a conventional manner the operationsof inverse cosine transformation, motion compensation, addition of theresidual in inter mode, etc.

[0059] We then go to a new subimage, step 4, during a new iteration.

[0060] This algorithm therefore makes it possible to construct just theimagettes whose index number is indicated in the list of imagettes to bedecoded, on the condition that they are coded in intra mode or coded inpredictive mode, the corresponding previous imagette having beenconstructed itself. If in this predictive mode, the correspondingimagette of the previous mosaic has not been reconstructed, it isnecessary to wait for a subsequent imagette coded in intra mode in orderto be able to construct it. It is therefore essential, during coding, toregularly code the imagettes in intra mode. It is the loop 12, 14, 13that makes it possible to check the coding in intra mode of a subimagewhen the overall image is coded in predictive mode. On exit from thisdecoding procedure one obtains a mosaic that is partially filled as afunction of the necessary imagettes.

[0061] When a subimage could not be constructed (step 15), one solutionconsists in utilizing the corresponding subimage of the previous mosaic.

[0062] An improvement of the invention consists in transmittingimagettes of different size. The process then consists, at the coderlevel, in coding, in addition to the image of the mosaic, the form ofthe support of the mosaic.

[0063]FIG. 4 represents, referenced 16, a form of mosaic for which theimagettes do not all have the same form. In this case, a coding of themosaic's support, referenced 17 in the figure, is performed. This codingis necessary only at the very start of the video session. However, amodification over time of the arrangement of the various imagettes ispossible. It is then necessary to update this support. The new supportis transmitted, in intra or inter mode, at each change of arrangement ofthe imagettes that gives rise to a change of support.

[0064] An implementation of the process utilizes the macroblock codingof a subimage. This is an exemplary embodiment and the macroblock can bereplaced by any image element without thereby departing from the fieldof the invention.

[0065] The process described can be extended to video objects of anyform, the support then having to be coded at each instant. Oneapplication relates to video games where the imagettes correspond tovarious cartoon-like characters.

[0066] It may be noted that if, at the encoding level, the addition ofresynchronization markers at the start of each subimage slightlyencumbers the binary stream, this escalation is limited and moreoveryields a resynchronization tool that facilitates the masking oftransmission errors.

[0067] The selecting of the subimages on the basis of thesynchronization markers can be done by assigning an index numbering tothe markers and to the subimages, an index number of a marker thencorresponding to a specified subimage. It can also be done by takingaccount of the sequential coding of the subimages, the nthsynchronization marker for the data relating to a mosaic image thencorresponding to the nth coded subimage of the mosaic.

[0068] Applications relate, among others, to interactive services usingthe video mosaic system. It may, for example, entail the consultation ofan electronic program guide where the user has the possibility ofexamining the various pages of a mosaic of programs.

1 Process for coding a mosaic of images of a sequence of images giving astream of data coded according to a syntax allowing the integration ofmarkers, for resynchronization so as to synchronize the stream onreception, the mosaic being composed of subimages, the subimages beingcomposed of image elements constituting any number of scan lines,characterized in that: the subimages (1 ₁ . . . 1 _(N)) are placed underone another in the image to be coded so as to give a mosaic, the mosaicis coded macroblock by macroblock according to a TV-type scan so as toprovide data of the stream, the first macroblock of each subimage beingidentified, a resynchronization marker is placed in the stream, beforeeach first element of a subimage, so as to identify the data relating tothe subimage. 2 Process according to claim 1, characterized in that thesynchronization markers are incremented from one subimage to the next. 3Process according to claim 1, characterized in that an image element isa macroblock, in that the coding utilizes the MPEG 2 standard and inthat the resynchronization markers are headers of slices. 4 Processaccording to claim 1, characterized in that an image element is amacroblock, in that the coding utilizes the MPEG 4 standard and in thatthe resynchronization markers are headers of groups of macroblocks. 5Process for coding according to claim 1, characterized in that, when thecoding of the subimages is in inter mode with motion estimation, themotion estimation utilizes a window for searching for the motion vectorwhich corresponds to the location of the subimage in the image. 6Process for coding according to claim 1, characterized in that thecoding of a subimage is forced to intra mode if it has not been coded inthis mode for a specified number of previous mosaic images. 7 Processaccording to claim 1, the subimages having different sizes (16), thesurface of these images constituting a mosaic support (17),characterized in that the mosaic support is also coded and the support'scoding data transmitted in the stream at the start of a video session. 8Process according to claim 7, characterized in that the mosaic supportvaries over time and in that the support's coding data are transmittedin the stream at each variation. 9 Process for decoding a mosaic ofimages on the basis of a stream of data comprising resynchronizationmarkers and data for coding mosaics of images consisting of subimages,characterized in that a filtering (6, 7, 8) of these data on the basisof the synchronization markers and a decoding are performed, to extractfrom the stream and to decode, from among the data relating to themosaics, only the data relating to selected subimages. 10 Processaccording to claim 9, characterized in that the filtering is dependenton the number of synchronization markers. 11 Process for decodingaccording to claim 9, characterized in that the decoded subimages arestored, and in that, if a subimage has been coded in inter mode, thecorresponding stored subimage of the previous mosaic is utilized for thedecoding. 12 Process according to claim 9, characterized in that, if asubimage may not be decoded, the corresponding subimage of the previousmosaic is taken into account. 13 Decoder for the decoding of a mosaic ofimages on the basis of a stream of data comprising resynchronizationmarkers and data for coding mosaics of images consisting of subimages,characterized in that it comprises a circuit (6, 7, 8) for filtering onthe basis of the synchronization markers and a decoding circuit (10) forextracting from the stream and for decoding, from among the datarelating to the mosaics, only the data relating to selected subimages.14 Reader of a video datastream transporting data for coding mosaics ofimages composed of subimages and resynchronization markers, for thedisplaying of a final image, comprising a decoder for the decoding ofthe stream and a processing circuit for the management of the decoder,characterized in that: the processing circuit sends the decoderinformation regarding the selected subimages useful for the displayingof the final image, the decoder comprises a circuit (6, 7, 8) forfiltering on the basis of the synchronization markers and a decodingcircuit (10), for extracting from the stream and for decoding, fromamong the data relating to the mosaics, only the data relating to theselected subimages.